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Foreword 

The  International  Network  of  Agencies  for  Health  Technology  Assessment  (INAHTA) 
has  been  formed  to  exchange  information  and  undertake  collaborative  activities  in 
health  technology  assessment.  At  the  INAHTA  Annual  Meeting  in  1997,  telemedicine 
was  identified  by  a number  of  agencies  as  appropriate  to  consider  through  an  INAHTA 
project. 

With  recent  advances  in  communication  technologies  and  the  increased  affordability  of 
equipment,  there  has  been  a rapid  worldwide  increase  in  telemedicine  projects  during 
the  1990s.  This  report  covers  issues  associated  with  the  introduction,  assessment,  and 
use  of  this  information  and  communication  technology.  It  is  intended  to  provide  a basis 
for  discussion  and  further  work  in  this  area. 

While  the  report  has  been  prepared  on  behalf  of  INAHTA,  the  views  expressed  are 
those  of  the  authors. 
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Summary 


• Telemedicine  is  an  information  and  communications  technology  which  shows 
promise  in  improving  quality  and  efficiency  of  health  care.  There  is  increasing  use 
of  telemedicine  in  many  countries. 

• However,  there  have  been  a number  of  impediments  to  creation  of  a fully  developed 
"telemedicine  society". 

• Those  seeking  to  introduce  telemedicine  will  need  to  give  general  consideration  to  a 
number  of  areas,  including  Population  and  Services,  Personnel  and  Consumers, 
Delivery  Arrangements,  Specifications  and  Costs,  so  that  a business  case  can  be 
made  for  acquiring  this  technology. 

• Assessment  of  telemedicine  applications  is  needed  to  assist  purchasing  and  planning 
decisions,  approaches  to  future  health  services  and  health  education,  and  also  to 
monitor  and  modify  the  use  of  the  technology  when  it  is  in  place. 

• Several  frameworks  for  assessment  have  been  suggested.  An  approach  is  outlined 
covering  the  domains  of  Technical  Assessment;  Effectiveness;  User  Assessment  of 
the  Technology;  Costs  of  Telemedicine;  Trials;  Economic  Evaluation  Methods;  and 
Sensitivity  Analysis. 

• A systematic  review  of  telemedicine  assessments  identified  29  studies  that  met 
criteria  for  reporting  outcomes  and  methodological  quality. 

• The  most  convincing  evidence  from  the  review  regarding  the  effectiveness  of 
telemedicine  deals  with  teleradiology,  teleneurosurgery,  telepsychiatry,  and 
transmission  of  echocardiographic  images.  Promising  results  have  also  been 
obtained  for  the  transmission  of  electocardiograms.  However,  even  in  these 
applications,  most  of  the  available  literature  refers  only  to  pilot  projects  and  short 
term  outcomes. 

• Further  scientific  assessment  studies  in  the  field  of  telemedicine  are  needed. 

Decision  makers  under  public  and  commercial  pressure  to  start  new  telemedicine 
services  should  link  introduction  of  new,  and  in  many  instances  costly  technology  to 
realistic  development  of  a business  case  and  subsequent  data  collection  and  analysis. 

• Decision  makers  should  also  note  that  an  assessment  of  a telemedicine  application 
will  be  strongly  influenced  by  the  context  in  which  it  is  undertaken.  Assessments 
will  typically  be  closely  linked  to  local  circumstances  and  their  results  may  not  be 
generalisable  to  other  situations. 
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Introduction 


Approach  used  in  the  report 

This  report  includes  two  main  sections.  In  the  first,  a discussion  on  general  principles  in 
the  assessment  of  telemedicine  applications  is  given,  together  with  some  background 
information  on  the  technology.  The  material  presented  draws  largely  on  the  experience 
of  the  author  agencies,  and  literature  that  they  have  found  to  be  valuable  in  developing 
approaches  to  assessment  of  telemedicine. 

The  second  part  of  the  report  is  a systematic  review  of  assessments  that  reported  the 
outcomes  of  telemedicine.  Administrative  changes,  patient  outcomes  and  economic 
assessment  were  considered. 

In  both  sections,  the  emphasis  is  on  the  assessment  of  telemedicine  in  specific 
applications. 

Definitions 

Telemedicine  is  the  use  of  information  and  communications  technology  to  provide 
health  care  services  to  persons  who  are  some  distance  from  the  provider.  It  includes 
technologies  such  as  facsimile,  medical  data  transmission,  audio-only  format  (telephone 
and  radio),  still  images,  and  full-motion  video  (32). 

Telemedicine  has  been  defined  in  different  ways.  The  US  Institute  of  Medicine 
considers  it  to  be  "the  use  of  electronic  information  and  communications  technologies  to 
provide  and  support  health  care  when  distance  separates  the  participants"  (40).  In 
considering  assessment  of  the  technology,  we  have  to  emphasize  the  definition  of 
Wootton  (77)  who  sees  telemedicine  primarily  as  a process,  rather  than  a technology. 
The  scope  of  telemedicine  as  a technology  is  considerably  wider  than  the 
telecommunications  equipment  and  systems  that  enable  exchange  of  information  at  a 
distance.  Telemedicine  should  be  regarded  in  terms  of  the  interaction  of  the  equipment 
and  the  information  transmitted  with  the  activities  of  the  health  care  professionals  who 
use  them,  and  the  consequences  for  patients  and  others  who  are  their  clients  (33). 

These  definitions  underline  the  role  of  the  telemedicine  as  a part  of  the  wider  process  of 
care  or  chain  of  care,  rather  than  a single  technology. 

Health  technology  assessment  (HTA)  provides  a suitable  framework  to  categorize 
and  estimate  the  various  effects  resulting  from  the  adoption  of  telemedicine,  and  to 
provide  a synthesis  of  these  as  input  to  future  decisions.  The  HTA  approach  compares 
alternatives  - in  this  case  telemedicine  versus  what  would  exist  in  its  absence.  A 
general  purpose  of  HTA  is  to  produce  information  about  the  effectiveness,  efficiency 
(cost-effectiveness),  and  safety  of  technologies.  In  addition,  it  provides  information 
about  the  ethical  and  social  aspects  of  their  use,  and  on  access  and  need  of  services  (9). 
Assessments  investigate  factors  related  to  the  adoption  and  diffusion  of  technologies 
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either  by  producing  new  information  or  by  analyzing  and  synthesizing  existing 
information.  Assessments  provide  health  technology  developers  and  manufacturers 
with  information  that  can  be  used  to  develop  technology  and  also  to  advertise  and  sell 
their  products. 

The  economic  success  of  health  care  operations  depends  on  the  efficiency  of  production 
and  the  fairness  and  equality  of  the  distribution  of  services.  Although  efficiency  is  often 
emphasized  in  health  care  assessments,  attention  should  also  be  paid  to  the  distribution 
of  the  services,  both  between  diseases  and  their  treatments  (74)  and  geographically.  The 
accessibility  of  health  services  is  an  important  issue  when  telemedicine  services  are  built 
up  in  a new  district  (40)  since  telemedicine  can  both  increase  the  utilization  of  special 
care  by  adding  a new  way  of  getting  services  and  decrease  the  need  for  secondary  care 
by  transferring  knowledge  to  primary  care  (53). 

A new  technology  has  to  be  proved  to  be  superior  to  the  technology  it  is  intended  to 
replace  (for  example,  more  effective  or  more  cost-effective).  This  is  of  considerable 
significance  in  telemedicine.  Telemedicine  may  have  favorable  attributes,  but  the 
existing  system  may  serve  the  population  well  and  also  be  capable  of  improvement. 
Comparison  should  include  the  present  (non-telemedicine)  system,  the  present  system 
when  upgraded,  and  the  telemedicine  alternative. 

The  telemedicine  alternative  should  be  sufficiently  mature  for  assessment  to  be 
meaningful.  Prototypes  or  telemedicine  practices  that  are  not  fully  integrated  with  the 
present  care  and  delivery  system,  can  be  assessed  in  pilot  studies.  However,  pilot 
studies  provide  mostly  interim  assessment  information,  which  gives  an  indication  of  the 
feasibility  of  a telemedicine  application. 

Telemedicine  technology 

Telemedicine  is  not  a new  concept.  For  many  years,  health  care  information  has  been 
sent  between  persons  at  different  locations  using  electronic  media.  Use  of  older 
approaches  (telephone,  fax)  is  commonplace.  However,  telemedicine  applications 
increasingly  utilize  the  newest  innovations  (both  in  hardware  and  software)  in 
computer  and  network  technologies.  In  addition,  the  technical  quality  of  the  other 
equipment  used  in  telemedicine  systems,  such  as  video  and  document  cameras,  is 
progressing  rapidly. 

The  consequence  of  these  developments  is  that  new  products  are  brought  into  the 
market  every  day  and  equipment  will  evolve  markedly  over  a period  of  years.  This  can 
be  seen  in  the  development  of  the  videoconference  (VC)  systems,  which  first  used  TV 
technology  (76)  and  then  TV  with  cable  or  satellite  transmission  (4).  The  TV  camera  has 
changed  to  video  technology  (such  as  roll'about  and  desktop  cameras)  and  the 
transmission  modalities  have  moved  to  different  commercial  telephone  lines  (ISDN  or 
ATM).  Also,  in  the  near  future,  the  Internet  and  radio  technologies  may  prove  to  be 
very  competitive  alternatives  to  the  current  networks.  This  continuing  shift  in 
technology  places  demands  on  those  seeking  to  introduce  and  operate  telemedicine. 


3 


Reid  (61)  points  out  that  "the  life  span  of  most  microprocessor-based  technologies  is  no 
greater  than  three  years.  The  products  themselves  last  longer,  but  in  that  short  time 
they  become  painfully  obsolete.  Some  telemedicine  technologies  are  more  capable  than 
others  of  being  upgraded  as  improvements  become  available.  'Face  lift'  upgrades, 
where  the  'old  box'  is  physically  and  completely  replaced  with  a 'new  box'  are  to  be 
avoided.  This  is  a very  expensive  undertaking.  Those  systems  that  can  be  upgraded  by 
loading  new  software  are  the  most  desirable.  Expansion  capability  is  a valuable  quality, 
particularly  in  telecommunication  transmission  equipment." 

In  teleradiology,  for  example,  there  may  be  a need  to  increase  the  bandwidth  of 
transmission  from  ISDN  lines  to  frame  relay  and  further  to  ATM  lines.  It  is  essential 
that  the  software  and  hardware  of  the  teleradiology  system  are  capable  of 
accommodating  increases  in  the  utilization  of  services. 
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Considering  the  need  for  telemedicine 

Purpose  and  effects  of  telemedicine 

The  implementation  of  telemedicine  is  partly  aimed  at  ensuring  that  health  care  services 
are  produced  at  an  optimum  location  from  the  viewpoint  of  patients,  health  care 
professionals,  and  funders.  For  example,  visits  to  hospital  outpatient  clinics  would  be 
unnecessary  when  the  treatment  or  advice  can  be  provided  in  a primary  health  care 
setting.  Telemedicine  projects  have  a bearing  on  the  distribution  of  specialized  health 
care.  However,  long  distances  are  not  a prerequisite  for  effective  telemedicine 
applications.  On  the  contrary,  telemedicine  can  be  used  even  over  short  distances,  such 
as  within  one  hospital  or  in  the  same  town  among  different  units  of  health  care  (11,  23, 
77). 

However,  each  case  must  be  considered  on  its  own  merits,  with  appropriate 
comparison  with  existing  technology  and  services.  The  coming  of  age  of  telemedicine 
has  instigated  public  health  care  programs  to  include  telemedicine  in  their  strategic 
plans.  For  example,  in  Finland,  many  Health  Care  Districts  have  set  goals  for 
telemedical  services  and  other  information  technologies  (56).  The  same  kind  of  strategic 
work  has  also  been  done  in  many  U.S.  States,  Canada,  Australia,  and  in  Northern 
Norway. 


The  objectives  of  telemedicine  can  be  specified  as  follows  (56): 

• to  reduce  direct  costs  to  the  health  care  sector  and  patients  and  to  reduce  indirect 
costs  (loss  of  production); 

• to  enhance  citizens'  equality  in  the  availability  of  specialized  medical  services  by 
bringing  these  services  to  remote  primary  health  care  centres,  or  equivalent  (access  of 
health  care  services); 

• to  improve  cooperation  between  specialized  and  primary  care  by  moving  services  and 
expertise  closer  to  citizens; 

• to  promote  the  proficiency  of  physicians  and  other  health  care  personnel  by  means  of 
teleconsultation,  and  video  conference-based  training  and  mentoring; 

• to  provide  at  least  the  same  level  of  clinical  care  to  patients  as  provided  by 
conventional  technology; 

• to  reduce  waiting  lists  for  specialized  health  care  by  providing  consultations  to  remote 
health  centres  and  educating  clinicians  to  perform  some  of  the  most  usual 
examinations  in  a specialty  (e.g.,  ophthalmology,  dermatology,  orthopedics); 

• to  improve  and  expedite  consultations  among  different  units  of  specialized  health  care 
in  acute  special  cases  (e.g.,  subspecialites  in  radiology  and  pathology,  and 
neurosurgery);  and, 

• to  increase  delivery  of  primary  and  secondary  health  services  to  the  home  and 
increase  access  to  health  care  information  directly  from  the  home. 
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There  are  great  expectations  for  the  future  of  telemedicine.  In  principle,  telemedical 
projects  are  considered  to  benefit  all  parties:  local  authorities  (cheaper  specialized 
services);  hospitals,  primary  health  care  centres  (improved  service,  increased  supply  of 
expertise);  patients  (changes  in  state  of  health  and  quality  of  life,  savings  in  costs  and 
time);  health  care  personnel  (increased  proficiency);  employers  (reduced  absenteeism 
from  work);  and  the  social  insurance  system  (reduced  reimbursements). 

In  addition,  private  medicine,  teleoperators,  and  firms  developing  and  producing 
telemedicine  equipment  have  big  expectations  about  business  opportunities  with  the 
technology.  However,  it  must  be  stressed  that,  for  most  applications,  these  remain 
expectations  which  require  validation  from  appraisal  of  appropriate  data. 

Impediments  to  the  development  of  a telemedicine  systems 

There  have  been  a number  of  impediments  to  creation  of  a fully  developed 
"telemedicine  society".  Consideration  of  these  factors  has  recently  been  undertaken  in 
Switzerland.  While  there  are  significant  geographical  and  societal  differences,  it 
appeared  that  the  Swiss  experience  had  similarities  to  that  in  Western  Canada,  for 
which  the  following  points  appear  to  be  significant  (1). 

• There  still  seems  some  lack  of  clarity  regarding  definitions  and  specifications  of 
telemedicine  - what  telemedicine  is  and  what  is  truly  needed  in  various  applications. 

• Much  of  the  early  focus  was  on  hardware  and  communications  details.  It  is 
necessary  to  consider  these,  but  it  has  taken  a while  for  there  to  be  full  appreciation 
of  the  need  to  consult  with  health  care  professionals  and  others  and  to  consider 
reorganization  of  services  and  infrastructure  that  are  consequential  to  adoption  of 
telemedicine. 

• The  majority  of  proposed  telemedicine  applications  have  not  progressed  beyond  the 
pilot  project  stage.  Economic  and  other  evaluation  of  most  applications  remains 
very  limited. 

• Some  potential  purchasers  and  users  of  telemedicine  are  not  comfortable  with  the 
short  life  time  of  many  equipment  components  and  the  speed  of  change  of  some 
technology. 

• A major  impediment  is  the  lack  of  resolution  as  to  how  physicians  (and  possibly 
other  health  care  professionals)  should  be  reimbursed  for  services  they  provide 
using  telemedicine.  This  may  be  more  signficant  in  private-oriented  health  care 
systems. 

• Another  dilemma  is  the  question  of  who  pays  for  establishment  of  infrastructure. 
There  is  competition  for  access  to  budgets. 
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• In  order  for  telemedicine  to  be  applied  in  a fully  effective  fashion  to  the  whole  health 
care  system,  there  will  be  a need  to  ensure  interconnectivity  between  different 
regions  and  institutions.  Standards  are  being  developed  but  these  have  taken  some 
time  and  will  not  necessarily  be  easily  implemented.  It  is  not  clear  all  technical 
issues  have  been  resolved,  or  that  it  will  be  easy  to  obtain  necessary  information 
from  suppliers. 

• Telecommunications  standards  are  not  consistent  across  the  health  care  system,  so 
there  is  variation  in  the  transmission  costs  and  capability  that  apply  to  various 
regions. 

• There  is  a concern  about  inconsistency  from  equipment  suppliers  regarding 
assurance  that  agreed  specifications  will  be  met.  Some  equipment  components  may 
remain  in  the  developmental  phase  for  longer  than  expected,  so  that  a telemedicine 
application  cannot  be  established  as  planned. 

• There  may  also  be  concern  regarding  provision  of  effective  routine  trouble-shooting 
support.  Operators  and  users  of  telemedicine  applications  require  assurance  that 
down  time  will  be  minimal. 

• Issues  relating  to  licensing  of  medical  staff  and  other  operators  are  not  resolved. 

• University-based  groups  have  research  interests  in  the  area  of  telemedicine,  but 
often  these  do  not  seem  to  be  closely  related  to  the  short  term  needs  of  the  health 
care  system. 

• There  are  concerns  that  the  introduction  of  telemedicine  might  lead  to  disruption  of 
established  referral  patterns,  linked  to  a possible  lack  of  control  of  health  care 
services.  These  range  from  individual  practices  concerned  with  loss  of  income  to 
broader  considerations  of  professional  bodies  regarding  the  future  pattern  of  health 
care. 

• There  is  still  a need  to  decide  on  requirements  for  real  time  applications  as  opposed 
to  store-and-forward,  which  may  be  entirely  adequate  in  many  situations. 

• Questions  of  availability  of  specialist  referral  advice  at  major  centres  are  not  fully 
resolved,  especially  for  real  time  applications.  Continued  negotiations  seem  needed 
on  scheduling  of  health  care  professionals  and  reasonable  funding  for  the 
organizations  concerned. 

• There  are  various  issues  related  to  relationships  between  different  levels  of 
government. 

• Health  authorities  have  faced  many  financial  and  administrative  pressures.  In  this 
sort  of  climate,  detailed  consideration  and  implementation  of  telemedicine  systems 
has  tended  to  be  pushed  aside  by  other  priorities. 
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• There  is  growing  acceptance  that  telemedicine  systems  require  assessment  and 
on-going  collection  of  relevant  data  for  administrative  purposes  at  the  local  level. 
However,  in  most  regions,  local  resources  for  these  tasks  are  minimal.  There  is  a 
real  problem  in  securing  resources  to  gain  some  understanding  of  the  effectiveness 
of  telemedicine  services.  The  worry  is  that  systems  might  be  acquired  but  then  not 
be  managed  efficiently  or  used  effectively. 

• Some  health  authorities,  in  part  because  of  their  proximity  to  major  centres,  see  no 
real  advantage  in  proceeding  with  telemedicine  systems  at  this  stage  (this  may  be  a 
legitimate  decision;  there  is  little  point  in  acquiring  the  technology  if  the  business 
case  is  poor). 

This  is  a diverse  collection  of  issues.  While  some  may  be  expected  to  resolve  without 
too  much  trouble,  others  may  pose  major  problems.  Overall,  users  need  to  have  a 
clearer  idea  of  their  requirements,  how  these  can  be  met  technically,  and  how  any  new 
services  can  be  organized  and  funded  on  an  on-going  basis. 

Making  the  case  for  telemedicine 

Prior  to  any  detailed  assessment,  those  seeking  to  introduce  telemedicine  will  need  to 
give  general  consideration  to  a number  of  points  so  a business  case  can  be  made  for 
acquiring  this  technology.  It  will  be  necessary  to  specify  clinical  requirements  and  the 
technology  being  considered  for  adoption.  The  specification  will  include  a clear  outline 
of  the  application  and  of  equipment,  staff,  and  other  resources.  Introduction  of 
telemedicine  will  raise  some  general  technical  issues.  There  will  be  a need  to  obtain 
assurance  that  available  hardware  and  software  can  provide  the  performance  required, 
at  a realistic  cost,  and  that  adequate  technical  support  will  be  available.  Validation  of 
specifications  and  of  performance  under  local  conditions  are  major  issues. 

To  realize  the  potential  benefits  from  telemedicine  through  time-related  gains  in 
efficiency,  planners  and  managers  will  need  to  put  in  place  changes  to  organizational 
structure  and  administrative  procedures.  Health  care  planners  and  managers  should  be 
informed  about  the  possibilities  for  future  use  of  telemedicine.  They  can  then  plan  and 
develop  the  health  care  delivery  system,  where  telemedicine  is  one  integrated 
component,  to  meet  regional  conditions  and  the  health  services  needs  of  the  population. 
Health  care  managers  and  planners  must  know  also  the  financial  resources  that  can  be 
used  for  telemedicine  in  the  future  (61).  Most  of  the  benefits  of  telemedicine  can  be 
realized  only  when  systems  are  widely  used  within  the  organization  and/or  between 
organizations.  That  is,  the  technology  is  accepted  by  patients,  health  care  professionals 
and  managers. 

Many  of  the  issues  on  use  of  the  technology  will  relate  to  changes  in  work  practices  and 
routines.  Active  consultation  with  all  staff  who  will  be  affected  by  introduction  of 
telemedicine,  and  use  of  their  expertise  in  developing  programs,  should  be  priorities. 
Availability  of  a person  to  take  responsibility  for  coordination  of  telemedicine 
applications  and  their  assessment  is  essential.  The  coordinator  must  have  a clear 
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understanding  of  the  overall  delivery  requirements  of  the  health  care  system  and  be 
responsive  to  the  needs  of  health  care  professionals  and  their  clients  (34). 

New  technology  can  also  offer  a better  way  of  sharing  and  utilizing  information.  In 
psychiatry  the  use  of  VC  has  increased  the  possibility  of  having  patient  care  planning 
negotiations  for  in-patients  before  they  go  back  to  their  remote  municipalities  (54). 

Yellowlees  (79)  suggests  that  management  and  support  of  telemedicine  projects  should 
be  from  the  bottom  up  rather  than  from  the  top  down.  Clinicians  may  be  more  aware 
about  the  technical  properties  of  the  telemedicine  alternative  and  thus  can  develop  the 
new  system  to  meet  the  needs  of  the  local  population.  However,  the  experiences  from 
Norway  suggest  that  the  importance  of  the  top  level  in  the  development  of  the 
telemedicine  system  increases  once  the  pilot  project  is  widened  to  other  sites  in  a region 
or  to  the  whole  country  (58).  The  telemedicine  applications  and  sites  should  be  selected 
pragmatically.  For  example,  a state- wide  telemedicine  system  might  be  built  up,  even 
though  some  regions  may  not  be  willing  or  able  to  participate  in  the  project  (79).  If  a 
region  does  not  have  required  telecommunication  facilities,  or  educated  health  care 
staff,  the  implementation  of  telemedicine  should  not  be  started  there. 

Consideration  of  the  project  specifications  should  help  to  determine  whether  a 
particular  telemedicine  application  is  appropriate  or  if  telemedicine  is  needed  at  all. 
Managers  should  be  seeking  strong  grounds  to  presume  that  the  proposed  application 
will  be  sustainable  (capable  of  providing  an  effective  service  continuously,  rather  than 
operating  only  as  a demonstration  project).  Details  will  be  required  of  the  equipment 
specifications  (to  eventually  be  verified  under  local  conditions),  the  method  of  data 
transmission,  maintenance  provisions,  and  training  arrangements  for  those  who  will 
use  the  technology.  A suggested  initial  checklist  is  shown  in  Table  1. 

The  management  process 

Telemedicine,  like  other  forms  of  information  and  communication  technologies, 
changes  the  management  process  in  health  care  (51).  It  is  important  that  the 
telemedicine  alternative  does  not  just  replace  or  automate  the  old  practice,  but  that  it 
renews  the  whole  health  care  process  in  question  (56).  The  changes  in  the  processes  do 
not  only  affect  health  care  costs  but  also  may  change  the  structure  of  personnel  used, 
legal  responsibilities,  and  the  place  and  nature  of  the  treatments.  Assessment  studies  in 
telemedicine  should  describe  in  detail  the  health  care  processes  used,  assess  how  well 
applications  worked  in  practice,  and  to  what  extent  process  and  outcomes  data  can  be 
generalized  nationally  and  to  other  health  systems. 

A report  by  CEDIT  has  outlined  a framework  for  analysis  of  telemedicine  applications 
in  the  context  of  hospital  services  (10).  Consideration  of  technical,  medical, 
economic/ cost,  organizational,  and  legal  aspects  is  recommended.  As  suggested  in  the 
overview  presented  here,  and  elsewhere  (31)  all  these  areas  will  require  attention  in 
planning  and  operating  a telemedicine  system. 


9 


Table  1 : Questions  to  consider  in  development  of  a business  case  for  telemedicine 


Population  and  Services 


What  applications  are  being  considered? 

• By  specialty. 

• By  administrative  task. 

What  are  the  current  delivery  arrangements  for  each 
specialty? 

• Approximate  level  of  demand. 

• Local  and  remote  health  care  providers. 

• Referral  arrangements. 

Personnel  and  Consumers 


Who  is  to  operate/use  the  telemedicine  application(s)? 

• Local  health  care  providers  - will  there  be  changes 
in  roles  and  responsibilities? 

• Remote  health  care  providers  - have  changes  in 
relationships  to  remote  providers  been  identified? 

Has  there  been  consultation  with  all  health  care  staff 
involved?  Is  there  acceptance? 

• Consider  views  of  all  staff. 

Should  there  be  wider  publicity  and  consultation 
regarding  the  telemedicine  services? 

• Consider: 

contact  with  patient  groups,  general  public, 
level  of  community  acceptance. 

What  training  programs  need  to  be  put  in  place? 

• Consider  qualifications  and  training  needs  for  all 
staff  who  will  be  involved  with  the  telemedicine 
application. 

Delivery  Arrangements 


How  many  sites  will  be  using  telemedicine? 

• Specify  the  applications  at  each  site. 

• Consider  sequency/timing  of  introduction  of 
telemedicine  at  each  site.  There  may  be 
advantages  in  phased  introduction. 

Has  scheduling  been  addressed,  at  least  at  a 
preliminary  stage? 

• Consider  scheduling  of  teleconsultation  sessions 
within  region/service. 

• Check/negotiate  availability  of  remote  providers. 

Is  real-time  telemedicine  essential  for  local  needs,  or 
might  store  and  forward  options  be  adequate? 

• Consider  if  immediate  availability  of  information  is 
important  for  clinical  and  administrative  needs. 

• Consider  factors  which  would  ensure  consistent 
real  time  services. 

What  are  the  storage  requirements  for  data  from  use 
of  telemedicine  applications? 

• Cost  and  flexibility  of  storage  requirements. 

What  back-up  arrangements  will  apply  should  the 
telemedicine  system  fail?  Have  data  security  and 
privacy  issues  been  considered? 

• Need  to  develop  contingency  plans. 
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Table  1 : Questions  to  consider  in  development  of  a business  case  for  telemedicine 

(cont'd) 


Specifications  and  Costs 


What  are  the  specifications  and  projected  costs  for 
purchasing  and  maintaining  telemedicine  equipment? 
Will  they  apply  fully  to  the  goods  that  are  to  be 
purchased? 

• Consider  how  specifications  will  relate  to  the 
application  in  question  and  the  needs  of  those 
using  it. 

• Ensure  availability  of  desired  equipment. 

• Bear  in  mind  that  equipment  will  need 
replacement,  perhaps  after  three  years. 

• Clarify  cost  and  details  of  maintenance 
arrangements. 

What  are  the  mode  and  costs  of  communication? 

• How  do  these  relate  to  expected  levels  of  use  of 
the  system? 

Who  is  going  to  invest  in  telecommunication  joint 
infrastructure? 

• Is  this  to  be  a responsibility  of  government  or  of 
operators? 

Will  the  telemedicine  application  cover  all  use  of  the 
service  in  question? 

• If  current  arrangements  are  to  stay  in  place  for 
some  cases,  consider  resource  and 
organizational  needs,  costs. 

How  will  changing  delivery  arrangements  affect  cost? 

• Changes  to  personnel  and  to  supplies  will  have 
consequences  for  costs. 

Are  there  other,  less  expensive  telemedicine  options? 

• Consider  potential  for  Web  use,  for  example,  in 
telemedicine-education  applications. 

• Other  telemedicine  approaches  (e.g.  telephone, 
secure  fax,  e-mail)  may  be  good  options  for  some 
applications. 

Have  issues  on  funding/reimbursement  for  use  of 
telemedicine  applications  been  resolved? 

• These  may  involve  wider  policy  matters. 

• Managers  and  users  will  require  assurance  on 
reimbursement  issues. 

• Consider  that  the  benefits  (e.g.  saved  travelling 
costs)  may  not  come  to  the  payer  of  telemedicine 
service. 

Source:  Reference  (34) 
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The  need  for  assessment 

Although  assessment  of  new  technologies  is  often  time-consuming  and  expensive,  there 
are  a number  of  reasons  for  critically  scrutinizing  telemedicine  programs.  As  noted  by 
Perednia  (60)  these  include  the  fact  that  health  care  resources  are  limited  and  data  are 
needed  to  compare  the  relative  value  of  telemedicine  systems  to  alternative  uses  of 
medical  resources.  Furthermore,  assessment  can  be  used  to  document  the  functional 
utility  of  telemedicine  as  a diagnostic  and  clinical  tool.  It  can  help  to  decide  whether 
remote  consultation  is  equivalent  to,  better  than,  or  worse  than  in-person  consultation. 
Assessment  can  also  provide  important  clues  as  to  why  a telemedicine  program  is 
successful  or  unsuccessful,  and  how  it  might  be  made  more  efficient.  Assessment  of 
telemedicine  applications  is  needed  to  assist  purchasing  and  planning  decisions,  and 
also  to  monitor  and  modify  the  use  of  the  technology  when  it  is  in  place  (34). 

The  example  of  the  heterogeneity  of  VC  technology  (4)  shows  that  there  is  no  unique 
telemedicine  technology  to  be  assessed.  The  soaring  demand  for  telemedical  equipment 
has  increased  the  need  for  health  technology  assessment  as  decision-makers  need  up-to- 
date  information  on  the  appropriate  uses  of  such  equipment  within  the  health  service. 
Older  assessments  of  equipment  and  networks  may  not  give  reliable  information  about 
current  products  in  telemedicine. 

Most  of  these  technologies  (equipment)  and  their  applications  are  by  now  sufficiently 
developed  to  be  suitable  for  scientific  evaluation  in  their  everyday  use.  However,  the 
activities  of  purchasers  and  providers  of  telemedical  services  (the  health  care  process) 
must  also  correspond  to  the  routine  operation  of  their  organisations;  otherwise  the 
effects  and  costs  of  the  telemedicine  services  cannot  be  reliably  estimated. 

The  introduction  of  telemedicine  applications  will  often  result  in  substantial  changes  to 
health  care  practices.  The  nature  of  individual  telemedicine  projects  will  vary  and  each 
case  will  need  to  be  considered  in  detail.  Investments  in  telemedicine  will  be 
accompanied  by  changes  in  patterns  of  care  - in  quality  of  service,  time,  and 
availability.  There  may  be  consequent  changes  in  health  outcomes  and  patient 
satisfaction.  Some  issues  will  be  relatively  specific  to  the  health  authority  or  other 
purchaser,  so  that  local  data  and  circumstances  will  need  to  be  considered. 

Continuous  assessment  is  required  to  appreciate  and  respond  to  changes  in  the  nature 
of  work  in  the  social  and  health  care  sectors.  Many  applications  of  telemedicine 
support  health  care  by  creating  an  infrastructure  that  is  conducive  to  efficient  and 
effective  teamwork.  Data  to  assess  the  function  of  the  technology  have  to  be  collected 
from  all  user  groups. 

The  implementation  of  telemedicine  requires  assessment  of  advantages,  disadvantages, 
costs  (especially  transaction  and  incremental  costs),  investment  schedules,  fluency  of 
communication,  need  and  access  to  services,  changes  in  work  processes,  and  the  process 
and  division  of  work.  Since  telemedicine  changes  the  conventional  division  of  work 
and  decision  making  of  clinicians,  the  legal  and  ethical  consequences  of  telemedicine 
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have  also  to  be  assessed  (34, 40,  56,  66),  though  studies  on  these  topics  have  been  slow 
to  emerge.  Definitive  assessment  of  a telemedicine  application  may  take  a considerable 
time  and  be  complicated  both  by  changes  to  the  technology  and  to  the  health  care 
system  in  question.  This  climate  of  continuing  change  will  often  suggest  the  need  for  a 
series  of  comparatively  rapid,  less  detailed  evaluations  to  provide  decision-makers  with 
timely  interim  advice. 

The  literature  indicates  that  technology  assessment  studies  dealing  with  telemedicine 
are  still  scarce.  The  use  of  telemedicine  usually  involves  significant  investments,  but 
decisions  on  these  are  often  not  informed  by  data  and  recommendations  from 
assessments.  A 1997  review  on  telemedicine  applications  was  able  to  identify  only  one 
cost-effectiveness  study  (51).  Since  then  several  other  economic  studies  have  been 
published  but  there  is  still  a major  need  for  high  quality  evaluation. 

Assessment  can  also  produce  data  on  the  optimum  timing  of  investments  against  the 
background  of  medium-term  structural  development  in  social  welfare  and  health 
services.  Telemedicine  offers  alternatives  to  older  methods  of  service  delivery. 
Cooperation  and  integration  of  services  between  the  social  and  health  care  sectors  can 
make  telemedicine  projects  economically  beneficial  earlier  than  when  investment  is 
made  independently. 

Technological  change  has  specific  implications  for  economic  research  on  social  welfare 
and  health  services.  Organizational  effectiveness  is  measured  by  the  momentum  of  the 
service  system,  businesses,  and  research  institutions  in  regional,  national,  and 
international  cooperative  networks.  Effects  should  be  assessed  on  the  basis  of  the 
functionality  (changes  in  processes),  fluency  (savings  in  time,  speed  of  diagnosis,  speed 
of  treatment),  and  efficiency  (cost-effectiveness)  of  each  telemedicine  project.  In  the 
future,  telemedicine  applications  which  may  be  connected  to  regional  or  nation-wide 
patient  registers  and  smart  cards  will  change  the  health  care  organizations  and  delivery 
systems.  In  that  environment,  assessments  of  telemedicine  should  consider  more 
aspects  of  the  information  flow  (transaction)  and  what  effects  this  has  on  the  clinical 
management  process  and  patient  outcomes. 
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Assessment  of  telemedicine 

General  considerations 

Assessment  is  targeted  at  the  health  care  process  (clinical  management  pathway).  The 
flow  of  information,  the  price  of  time  used  in  traveling,  waiting  time  for  consultation, 
and  traveling  costs  are  important  areas  for  analysis.  It  is  important  to  assess  how  the 
new  technology  will  change  the  work  processes  in  health  care.  In  diagnostic 
applications  (e.g.,  teleradiology)  telemedicine  can  make  possible  new  services  to  remote 
centers.  However,  the  measurement  of  the  costs  and  outcomes  of  diagnostic 
technologies  can  be  challenging  since  they  will  commonly  have  an  indirect  effect  on 
outcomes. 

While  these  are  useful  guiding  principles,  telemedicine  systems  vary  considerably  in 
their  complexity.  The  nature  of  the  application  may  affect  the  scope  of  the  assessment. 
For  example,  there  will  be  differences  between  assessment  of  a videoconference  facility 
(VC)  and  a hospital-wide  PACS  (Picture  Archiving  and  Communication  System).  While 
the  assessment  of  VC  can  be  made  even  within  one  sub-group  of  a specialty  (e.g., 
orthopedics),  PACS  applications  require  a much  more  holistic  approach  to  the  whole 
hospital  and  to  the  information  flow  between  wards  and  other  actors  within  it. 

There  is  a need  to  consider  the  assessment  of  each  application  in  an  individual  way. 

The  nature  of  the  assessment  will  depend  on  the  context  in  which  it  takes  place. 
Context-specific  features  may  include  sources  and  scope  of  data,  organisation  of 
services,  case  mix  and  social  values.  Economic  assessment  should  account  for  context 
variations  in  issues  such  as  the  choice  of  comparators,  values  on  costs  and  consequences 
and  the  financing  perspectives  considered. 

Special  characteristics  of  telemedicine  as  a target  of  assessment 

Several  publications  have  discussed  telemedicine  as  a target  of  assessment  (34,  40,  51, 

53,  56).  One  problem  that  is  commonly  referred  to  is  the  rapid  advance  of  information 
and  telecommunication  technologies,  which  can  make  the  assessments  of  current 
telemedicine  applications  out-of-date  even  before  they  are  available  for  decision 
makers.  The  choice  of  telemedicine  technology  for  assessment  is  then  very  important, 
as  discussed  earlier.  Feasibility  studies  can  be  used  to  detect  good  candidates  for  fuller 
assessments. 

The  prices  of  telemedicine  equipment  are  high  after  technologies  enter  the  market,  but 
decrease  rapidly  once  they  are  widely  used.  Assessment  studies  may  take  several  years 
and  the  cost  structures  can  change  considerably  during  the  assessment  period.  This 
effect  can  partly  be  controlled  by  using  the  latest  purchasing  prices  of  the  equipment 
and  by  presenting  sensitivity  analysis  of  the  influence  of  the  expected  changes  in  costs. 

The  size  of  sites  providing  the  telemedicine  link  can  range  significantly  between  regions 
and  countries.  The  number  and  size  of  clients  (e.g.,  primary  health  care  centres) 
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influence  both  the  way  work  is  organized  and  costs  in  the  special  care  site.  After  a pilot 
project,  there  will  be  an  expectation  that  the  service  should  be  made  available  to  other 
clients  without  big  changes  in  costs  or  effectiveness  - that  is,  the  results  of  the  feasibility 
study  will  be  generally  useable.  Ideally,  an  assessment  study  should  present 
information  for  clients  with  different  levels  of  service  utilization  and  for  service 
producers  with  different  kinds  of  clients.  In  addition,  telemedicine  may  produce 
changes  in  health  care  processes  that  may  have  effects  on  health  care  organizations  in 
the  long  run.  This  can  change  the  outcome  and  costs  of  the  alternative  to  telemedicine. 
Since  most  studies  to  date  have  been  made  on  pilot  projects,  the  assessment  results  are 
difficult  to  interpret  for  longer  term  decision  making  (23). 

Telemedicine  applications  are  mostly  new  and  there  may  not  be  solid  legal  and  ethical 
rules  for  their  use  in  health  care.  These  questions  should  be  discussed  and  accepted  by 
ethics  bodies  of  the  organizations  (sometimes  even  at  a national  level)  before  an 
application  is  properly  assessed.  Confidentiality  and  security  of  information  are 
common  concerns.  It  should  be  remembered  that  in  existing  systems  there  is  potential 
for  some  patient  information  to  fall  into  the  hands  of  outsiders  (79).  However,  in 
telemedicine  the  danger  of  substantial  amounts  of  information  falling  into  the  wrong 
hands  is  much  bigger  without  an  adequate  security  system  in  place.  Security  and 
privacy  issues  should  be  considered  in  planning  and  assessment  of  telemedicine 
applications. 

Telemedicine  applications  may  be  difficult  to  evaluate,  since  study  designs  that  will  be 
feasible  for  some  operators  may  not  meet  high  scientific  standards.  Randomization  and 
double  blinding  may  not  be  possible  in  many  studies.  However,  controlled  trials  can  be 
made  in  nearly  all  telemedicine  applications.  Although  the  HTA  literature  tends  to 
focus  on  quantitative  methods,  qualitative  methods  can  produce  important  information 
about  the  meaning  in  human  or  organizational  transactions  and  in  social  and  cultural 
context  (51). 

According  to  Ferguson  and  Keen  (21),  information  and  communication  technology  has 
a profound  impact  on  the  modern  health  care  system;  this  impact  should  be  evaluated 
at  the  level  of  the  entire  system.  Transaction  costs  are  a frame  of  reference  in  evaluating 
the  relationship  between  information  and  communication  technology  and  the  efficiency 
of  services  (21).  The  core  of  transaction  costs  lies  in  imperfect  information.  Insufficient 
or  imperfect  information  increases  costs.  Transaction  costs  can  be  reduced  by  means  of 
information  and  communication  technology  but  the  important  issues  are  the  optimum 
level  of  investment  and  who  is  to  pay  for  this.  At  issue  are  external  effects  which  are 
positively  influenced  by  fluent  communication. 

Glandon  and  Buck  (29)  showed  that  economic  assessments  of  information  and 
communication  technology  have  yielded  very  little  information  about  the  real  costs  and 
benefits  of  investments.  The  same  conclusion  is  frequently  made  in  many  fields  of 
telemedicine  (4,  53,  68,  79). 
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Telemedicine  can  contribute  to  integrated  communication  systems  that  allow 
information  to  be  disseminated  to  greater  numbers  of  health  care  professionals.  For 
example,  PACS  installations  that  are  used  to  deliver  and  store  different  kind  of  images, 
can  be  connected  to  telemedicine  networks  to  work  interactively  between  primary  and 
secondary  care  (63,  68).  This  is  a step  towards  regional  and  global  networks  and  patient 
records  that  are  planned  to  increase  the  cost-effectiveness  of  health  care  in  the  future 
(21).  However,  the  assessment  of  large  information  systems  in  medicine  will  be  a 
complex  undertaking. 
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Frameworks  for  assessment 

Ideally,  assessment  should  provide  a broad  description  of  telemedicine  for  decision 
makers,  covering  technical,  clinical,  economic,  ethical,  legal  and  organizational  issues. 

In  practice,  assessments  have  been  constrained  by  availability  of  data,  timing  of  policy 
and  administrative  decisions,  shortage  of  evaluators,  and  inertia  within  health  care 
systems.  Guidance  for  assessment  has  been  provided  in  frameworks  formulated  by  a 
number  of  authors  (34,  40,  51,  53,  56,  64).  These  make  provision  for  detailed  evaluation 
of  a number  of  attributes,  though  to  date  much  more  limited  assessment  has  been 
undertaken  in  practice.  In  the  discussion  presented  here,  particular  reference  is  made  to 
the  framework  summarized  in  Table  2,  which  draws  on  work  undertaken  by  FinOHTA 
(56). 

Estimation  of  the  outcomes  of  telemedicine 

The  following  general  hierarchy  for  evaluation  of  diagnostic  applications  of 
telemedicine,  following  that  of  Fineberg  et  al.  (24),  gives  an  indication  of  stages  to  be 
considered  in  addressing  the  efficacy  and  effectiveness  of  the  technology: 

1.  technical  accuracy  of  the  images  or  other  data; 

2.  diagnostic  quality  of  images  or  other  data  (e.g.,  sensitivity  and  specificity  of  the 
diagnoses,  receiver  operating  characteristic  (ROC)  analysis); 

3.  diagnostic  effectiveness  (changes /confirmations  of  diagnoses); 

4.  therapeutic  effectiveness  (changes  in  clinical  management  of  patients);  and 

5.  changes  in  health  status  of  the  patients  including  quality  of  life  (HRQOL). 

A complete  assessment  of,  for  example,  teleradiology  services  between  primary  and 
secondary  care,  requires  discussion  of  all  these  questions  (39).  An  analogous  list  could 
be  formulated  for  therapeutic  applications. 

Technical  quality:  Stage  1 concentrates  on  the  technical  quality  of  the  method.  This  will 
include  the  technical  quality  of  the  teleradiological  images  after  a transmission,  whether 
the  specialist  could  use  different  image  processing  properties,  and  whether  the  transfer 
of  all  images  (or  other  data)  was  made  successfully.  Technical  accuracy  studies  are 
feasibility  studies  which  show,  mostly  in  laboratory  circumstances  or  in  pilot  projects, 
that  a certain  telematic  transmission  is  technically  possible,  and  that  the  method  fulfills 
technical  quality  requirements.  Some  of  these  issues  may  be  picked  up  through  routine 
quality  assurance  protocols,  and  in  the  development  of  a business  case  considered 
earlier. 
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Table  2:  Assessment  of  telemedicine  between  a remote  place  Primary  Care  (PC)  unit 

and  Secondary  Care  (SC)  units  (or  other  special  care  unit). 


Items 

Symbols 

Measurement  topic 

Technical  Assessment: 

+ technical  quality  of  image  (and  voice)  transfer 

time,  luminance,  resolution 

+ reliability,  validity  and  other  characteristics 

sensitivity,  specificity,  ROC,  rate  of 
successful  transmissions 

Effectiveness, 

+ diagnostic  quality 

E 

sensitivity  and  specificity  of  diagnosis 

+ changes  in  health  related  quality  of  life 

E,  U 

e.g.,  QALY,  HRQOL  profiles 

+ clinical  changes  in  health 

E 

survival,  symptoms 

+ changes  in  management  process  in  hospital 

clinical  pathway,  improved  quality  of  care 

+ increasing  know-how  in  PC 

decreased  utilization  of  consultations, 
improved  quality  of  primary  care 

+ non-health  outcomes  of  a patient 

certainty,  access  of  care,  equality 

User  Assessment  of  the  Technology: 

+ doctors 

user  questionnaire  from  quality,  usability 
and  satisfaction 

+ nurses  and  other  staff 

user  questionnaire  from  quality,  usability 
and  satisfaction 

+ patients  (if  relevant) 

user  questionnaire  from  quality,  usability 
and  satisfaction 

Costs  of  Telemedicine: 

+ investment  costs  in  PC  and  SC 

FC 

direct  cost;  e.g.,  equipment  and  network 

+ monthly  user  charge  of  equipment  in  PC  and 
SC 

FC 

direct  cost;  rental  cost  of  lines,  maintenance 
(10-15%  of  capital  costs) 

+ costs  of  used  communication  lines  in  PC  and 
SC 

VC 

direct  cost;  user  charge  per  hour 

+ wages  of  doctors  and  other  staff  in  PC  and 
SC 

VC 

direct  cost;  time  used  on  telemedicine. 

+ education  of  the  technology  (PC  and  SC) 

VC 

direct  cost;  education  of  personnel  and 
support  services 

+ other  relevant  costs  in  PC  and  SC 

FC  or  VC 

direct  cost;  room,  energy,  administration 

+ costs  of  a patient  (and  close  relatives) 

direct  cost;  travelling  and  domestic  costs, 
other  medical  costs 

+ lose  of  working  and  leisure  time  of  a patient 

indirect  cost;  time  away  from  work 

+ intangible  costs 

indirect;  value  of  death,  pain  and/or 
worsened  HRQOL 

Total  Costs  (TC)=  Fixed  Costs  (FC)  + Variable  Costs  (VC)  + other  direct  costs 
(+  possible  indirect  costs): 
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Table  2:  Assessment  of  telemedicine  between  a remote  place  Primary  Care  (PC)  unit 

and  Secondary  Care  (SC)  units  (or  other  special  care  unit)  (cont'd). 


Items 

Symbols 

Measurement  topic 

Trials: 

+ randomisation 

most  preferred 

+ before  and  after  comparison  within  hospital 

+ control  groups 

(+  experimental  or  nonexperimental  analysis) 

Economic  ] 

Evaluation  Methods: 

+ cost  or  effectiveness  analysis  (CA,  CMA,  EA) 

TC  or  E 

Only  TC  or  E measured 

+ cost-benefit  analysis  (CBA) 

TC-E, 

TC/E 

Both  TC  and  E in  monetary  units 

+ cost-effectiveness  and  -utility  analysis 
(CEA,  CUA) 

TC/E, 

TC/U 

E in  natural  units  or  preferences  and  U in 
utilities 

Sensitivity  Analysis: 

+ distance 

distance  between  co-operative  sites 

+ number  of  patients 

patient  load  in  both  sites 

+ duration  of  the  investment 

years  of  utilization 

+ effects  of  changes  in  technology  in  the  future 

e.g.,  improved  technical  quality  and 
shortened  duration  of  transmissions 

Source:  Reference  (56) 

CA  = Cost  analysis  CMA  = Cost  minimisation  analysis  E=Effectiveness 

EA=Effectiveness  analysis  HRQOL=Health  related  quality  of  life  QALY=Quality  adjusted  life  year 

ROC  analysis=Receiver  operating  characteristic  analysis  TC=Total  costs 

U=Utility 
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The  technical  properties  influence  the  sensitivity  and  specificity  of  the  telematic 
diagnostic  method  compared  to  a standard  method  (gold  standard,  if  it  exists).  One 
commonly  used  method  to  test  the  diagnostic  quality  is  to  perform  an  ROC  analysis 
where  the  diagnostic  accuracy  of  telematically  transmitted  images  is  compared  to  that 
of  the  same  images  sent  by  a conventional  method  (e.g.,  plain  film  sent  by  post).  The 
use  of  ROC  analysis  in  telemedicine  is  explained  and  reviewed  by  Taylor  (68)  and 
Ruggiero  (63).  Aspects  of  technical  accuracy  and  diagnostic  quality  require  evaluation 
and  resolution  before  consideration  is  given  to  assessment  of  clinical  effectiveness. 

Clinical  effectiveness:  The  measurement  of  clinical  effectiveness  of  telemedicine  is  one  of 
the  prime  objectives  in  assessment  studies.  If  information  on  effectiveness  is  poor,  other 
types  of  assessment  (e.g.,  economic  or  social)  are  unlikely  to  give  reliable  or 
generalizable  results.  Clinical  health  status  can  be  measured  with  the  standard 
measures  used  in  each  specialty.  The  most  commonly  available  clinical  measures  are 
the  mortality  and  morbidity  rates  (Table  2).  Many  telemedicine  applications  aim  to 
produce  at  least  the  same  clinical  outcome  as  alternative  conventional  interventions  and 
the  differences  in  mortality  or  indications  of  morbidity  (e.g.  duration  of  sick  leave)  are 
often  very  small.  However,  in  some  applications  (such  as  teleneuromedicine)  the  aim 
may  be  to  improve  clinical  outcomes  as  a result  of  the  technology,  in  part  through  more 
adequate,  faster  transfer  of  the  patient. 

As  with  other  diagnostic  technologies,  evaluation  data  are  more  readily  obtained  for 
levels  3 and  4 of  the  list  given  above  (diagnostic  and  therapeutic  effectiveness)  than  for 
changes  in  health  status.  Those  taking  decisions  on  telemedicine  will  commonly  have 
to  rely  on  surrogate  measures  of  effectiveness  which  may  have  tenuous  links  to  health 
status  indicators. 

Diagnostic  effectiveness  refers  to  how  much  a diagnostic  method  affects  the  clinician's 
thinking  about  the  disease  or  condition.  It  can  be  estimated,  for  example,  by  calculating 
the  log  likelihood  ratio  from  the  diagnosis  before  and  after  use  of  imaging  technology 
(39).  Such  studies  will  tend  to  be  most  often  undertaken  at  a pilot  project  stage  and 
could  often  be  regarded  as  being  part  of  establishing  the  feasibility  of  a telemedicine 
method. 

With  therapeutic  effectiveness,  appraisal  of  the  influence  of  telemedicine  on  patient 
management  will  often  continue  beyond  the  pilot  project  stage.  Users  and  funders  of 
telemedicine  services  may  wish  to  undertake  on-going  monitoring  of  patient 
management,  possibly  linking  such  measures  to  quality  assurance  programs.  In  the 
case  of  diagnostic  imaging  technology  there  will  be  interest,  as  with  stage  3,  in  the 
influence  of  telemedicine  on  diagnosis,  monitoring /staging  and  excluding  diseases  or 
conditions. 

Health  outcomes:  In  stage  5 of  the  assessment,  in  principle,  the  outcome  of  the  whole 
management  process  is  measured,  that  is  changes  in  health  status  of  the  patients  and  in 
their  Health-Related  Quality  of  Life  (HRQOL)  (Table  2). 
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Measurement  of  changes  to  patient  outcome  poses  major  challenges  to  those  assessing 
telemedicine  applications,  for  reasons  discussed  elsewhere  in  this  report.  In  general: 

• availability  of  administrative  and  other  data  for  conventional  services  may  be 
limited; 

• differences  in  outcomes  between  the  telemedicine  and  conventional  options  may  be 
modest,  while  numbers  of  subjects  may  be  small  and  the  power  of  the  study  low; 

• there  may  be  changes  to  telemedicine  technology;  and 

• there  may  be  changes  to  the  conventional  (non-telemedicine)  technology  and  to  the 
structure  of  the  health  care  systems. 

Sisk  and  Sanders  (64)  have  noted  that  intermediate  outcomes  may  be  acceptable 
measures  of  health-related  effects. 

Study  design  in  telemedicine  assessment:  The  strength  of  evidence  of  a telemedicine  study 
depends  to  a large  extent  on  the  study  design  (Table  2).  The  starting  point  is  that  there 
should  be  a comparison  between  a new  and  some  conventional  alternative(s).  The 
study  can  then  be  designed  according  to  protocols  used  widely  in  clinical  and 
epidemiological  studies,  with  comparative  strength  of  study  indicated  by  classifications 
such  as  those  of  Jo  veil  and  Navarro-Rubio  (43).  Randomized  controlled  trials  (RCTs) 
with  large  sample  sizes  offer  the  strongest  evidence  for  decision  making.  However,  in 
telemedicine  studies,  randomization  may  be  difficult  or  impossible. 

A further  limitation  on  the  use  of  RCTs  in  telemedicine,  as  argued  by  McDonald  (52)  is 
that  in  reality  they  will  be  undertaken  only  during  the  pilot  or  early  routine  use  of  a 
telemedicine  application.  As  with  other  health  care  technologies,  RCTs  will  be  used  in 
telemedicine,  if  at  all,  to  assess  the  efficacy  of  the  technology.  Such  RCTs  may  not  be 
widely  generalisable  to  other  health  care  systems  and  will  not  necessarily  be  more  than 
a general  guide  to  the  effectiveness  of  the  telemedicine  application,  should  it  prove  to 
be  sustainable  and  move  into  long  term  routine  use. 

The  reality  in  the  evaluation  of  telemedicine  is  that  weaker  study  designs  may  have  to 
be  used.  As  a result,  decision  makers  must  be  cautious  in  the  degree  of  inference  from 
the  assessment  results  published  in  the  field.  Telemedicine  assessments  have  so  far 
been  on  stronger  ground  in  considering  effects  of  the  technology  on  time-related 
consequences  of  health  care  services  and  on  organizational  issues. 

Quality  of  life  measures:  Measures  of  HRQOL  in  an  assessment  study  in  telemedicine 
will  depend  on  the  aim  of  the  study  and  the  disease  or  condition  being  considered.  For 
example,  if  the  study  assessed  the  effectiveness  of  a telepathology  system  in  breast 
cancer  surgery,  some  cancer-specific  instrument  can  be  used  to  measure  QOL  outcomes. 
However,  the  relatively  short-term  intervention  of  most  telemedicine  applications  and 
the  indirect  nature  in  health  effects,  are  impediments  to  this  kind  of  long-term  outcome 
measurement.  In  an  application  such  as  teleradiology,  where  many  types  of  conditions 
are  covered,  a disease-specific  instrument  cannot  be  used.  Generic  measures  developed 
to  be  applicable  in  most  diseases  may  be  more  useful.  At  least  in  the  short  term,  QOL 
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outcomes  may  be  more  likely  to  relate  to  time-related  aspects  such  as  differences  in 
waiting  times  or  travel  requirements  consequential  to  use  of  the  new  technology. 

Useful  indications  of  possible  effects  of  QOL  may  be  obtained  from  preliminary  surveys 
during  feasibility  studies  (17). 

The  descriptive  system  of  HRQOL  can  be  related  to  values  that  a population  has  from 
health  states  (7, 19, 47,  75).  The  scaling  system  of  the  generic  measures  separates  the 
preference  weighted  single  index  measures  to  their  own  category,  that  is,  utility 
measures  (U).  They  can  be  used  in  the  utility  analysis  where  the  outcomes  are  shown  as 
a combination  of  time  and  change  in  HRQOL,  in  terms  of  quality  adjusted  life  years 
(QALYs).  Incorporation  of  QALY  measurements  into  telemedicine  assessments  has  yet 
to  occur  and  methodological  issues  would  need  to  be  carefully  considered. 

Other  Outcomes:  In  addition  to  process  oriented,  clinical  and  HRQOL  outcomes, 
telemedicine  may  also  have  substantial  educational  effects  in  primary  care  (Table  2). 

For  example,  during  videoconference  consultations  a specialist  can  guide  a GP  to  make 
new  types  of  examination  on  a patient.  The  GP  can  hear  about  new  treatments  or  other 
medical  information.  This  kind  of  positive  externality  benefits  both  the  patient  in  the 
consultation  and  also  the  future  patients  of  the  GP.  Although  assessment  of  the 
educational  effect  of  telemedicine  is  difficult,  it  should  be  included,  whenever  possible, 
in  the  projects  where  doctors  on  different  levels  of  the  specialty  or  from  different 
specialties  communicate  with  each  other.  Bergmo  (5)  has  included  educational  effects 
in  an  economic  analysis  of  teleconsultation  in  otorhinolaryngology.  The  educational/ 
training  effects  of  telemedicine  may  have  an  influence  on  the  sustainability  and  costs  of 
a specific  application  as  it  moves  from  the  pilot  project  phase  into  routine  care  (such 
effects  may  tend  to  decrease  use  of  telemedicine).  A contrary  trend  is  that  establishing 
of  telemedicine  may  lead  to  previously  unmet  demand  for  services  (35). 

Telemedicine  consultations  can  also  serve  as  a method  of  quality  assurance  for 
diagnostic  procedures.  For  example.  X-ray  images  can  be  sent  for  further  examination 
(or  second  opinion)  and  the  results  compared  to  the  original  diagnostic  decision  and 
subsequent  management  decision.  The  feedback  then  improves  the  overall  quality  of  a 
radiology  unit.  The  increased  certainty  of  diagnostic  information  is  an  important 
outcome,  since  it  reflects  the  overall  patient  management  process  and  also  the  quality  of 
process  of  care. 

The  implementation  of  telemedicine  can  reduce  the  time  required  to  get  a service 
(access).  Influence  on  time  of  transactions  may  impact  on  both  short  and  longer  term 
health  effects  to  the  patient,  and  'non-health'  effects  connected  to  increased  information 
such  as  certainty  of  diagnosis  and  the  value  of  time.  In  principle,  the  value  of  these 
non-health  outcomes  to  patients  can  be  estimated  using  a willingness  to  pay  method  or 
conjoint  analysis  (53).  However,  such  assessment  approaches  have  yet  to  be 
implemented  in  telemedicine  and  their  validity  and  generalisability  would  need  to  be 
established. 
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Quality  and  user  satisfaction 

Quality  problems  with  telemedicine  might  include: 

• overuse  of  care  (e.g.  unnecessary  use  of  telemedicine); 

• underuse  of  care  (e.g.  failure  to  refer  a patient  for  necessary  services);  and 

• poor  technical  or  interpersonal  performance  (e.g.  failure  to  get  proper  voice  or 
image  in  videoconferencing,  and  difficulties  in  interviewing  during  telesurgical 
consultation). 

Each  type  of  problem  is  important.  The  first  two  are  connected  to  health  policy  making 
in  a broader  sense,  since  they  are  closely  connected  to  equity  questions  and  the  funding 
of  health  care.  Technical  quality  is  not  only  the  quality  of  images  or  other  transmitted 
information,  but  also  the  usability  of  software  and  hardware. 

Users'  satisfaction  with  the  facilities  is  particularly  important  when  adopting  new 
alternative  technologies  (9,  34).  Staff  who  are  dissatisfied  with  the  equipment  will  not 
use  it.  In  addition,  satisfaction  of  patients  and  their  families  should  be  ascertained. 
Further,  it  should  be  noted  that  patients'  satisfaction  with  telematic  services  is  also 
influenced  by  ancillary  operations,  such  as  the  integration  of  doctors'  appointments  and 
diagnostic  tests.  User  and  patient  satisfaction  will  be  an  important  aspect  for 
assessment  at  the  pilot  project  stage,  using  survey  methodology.  Follow-up  surveys,  as 
the  technology  comes  into  routine  use,  will  also  be  desirable.  Decisions  informed  by 
such  surveys  will  commonly  be  those  for  the  operators  as  much  as  for  the  funders  of 
telemedicine. 

Estimation  of  the  costs  of  telemedicine 

From  whose  viewpoint  should  costs  be  examined?  This  question  of  perspective  is  very 
important  in  every  assessment  study,  since  an  item  may  be  a cost  from  one  point  of 
view,  but  not  from  another.  For  example,  the  travel  of  a patient  to  a health  facility  will 
be  associated  with  costs  from  the  patient's  perspective  and  from  a societal  point  of  view 
but  not  from  the  perspective  of  the  Ministry  of  Health  (19,  33).  Usually,  costs  accrued  to 
each  party  in  the  project  will  be  taken  into  account  (producer  of  the  health  service, 
patient,  patient's  family,  other  services,  the  rest  of  society).  If  the  project  is  not  aimed  at 
a specific  perspective,  assessment  studies  utilize  the  societal  point  of  view,  that  is,  all 
relevant  costs  are  estimated.  If  some  cost  items  are  common  in  all  compared 
alternatives,  they  may  be  left  out  of  the  study. 

Table  2 shows  the  check  list  of  the  costs  that  are  applicable  in  most  telemedicine 
projects.  Once  the  use  of  resources  in  natural  non-monetary  units  have  been  measured, 
the  quantity  units  of  resources  are  multiplied  by  their  monetary  values.  The  separation 
of  used  resources  and  their  monetary  values  is  important  for  inter-country 
comparisons,  since  it  enables  the  re-adjustment  of  the  cost  data  between  different  health 
care  systems  and  price  levels  (9). 
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In  the  valuation,  market  prices  should  be  used.  If  some  prices  are  subsidized  (e.g.  those 
of  laboratory  or  radiological  examinations  within  a hospital),  they  should  be  readjusted 
to  correspond  to  their  market  prices.  The  economic  definition  of  costs  also  includes  the 
notion  that  the  cost  estimates  obtained  from  accounting  may  not  reflect  the  real 
opportunity  cost  of  used  resources.  For  example,  buildings,  land,  and  equipment  may 
have  been  written  off  in  the  accounting,  but  will  have  opportunity  cost  in  the  use  of 
other  projects  (9).  To  make  the  comparison  between  different  studies  easier,  the  costs 
should  be  valued  using  standard  cost  values  which  reflect  the  central  tendency  of  cost 
values  of  the  used  resources. 

The  time  horizon  of  the  cost  analysis  may  influence  to  the  cost  estimates  of  project 
alternatives.  The  telemedicine  application  and  the  alternatives  with  which  it  is 
compared  may  have  different  distributions  in  time  of  costs  and  benefits.  Higher  cost 
today  may  reduce  costs  in  the  future.  In  many  telemedicine  projects,  it  has  been 
assumed  that  while  current  investment  costs  can  be  high,  the  reduction  in  prices  in 
future  together  with  decreased  need  for  secondary  care  (by  the  educational  effect  on 
primary  care)  will  significantly  decrease  the  lifetime  costs  of  a project.  All  future  costs 
are  discounted  to  their  present  value  by  using  the  appropriate  social  discount  rate. 

Types  of  cost:  Costs  are  divided  into  three  general  classes:  direct,  indirect  and  intangible 
costs.  Costs,  estimated  as  cost  flows,  are  affected  by  the  technology  used,  the 
functionality  of  the  health  care  process,  and  the  changes  in  the  care  provided.  Most 
telemedicine  projects  are  aimed  at  reducing  the  expenditure  related  to  the  transaction  of 
the  specialized  medical  services.  In  telemedicine,  information  such  as  x-ray  images 
travel  between  primary  care  (PC)  and  secondary  care  (SC)  units,  as  an  alternative  to 
travel  by  the  patient  or  by  the  specialist  between  these  centres.  In  the  cost  assessment, 
both  the  conventional  and  telemedicine  alternatives  have  to  be  considered  in  elements 
that  can  be  determined,  for  example,  from  the  actual  clinical  management  pathway. 

Direct  costs  can  occur  either  within  health  care  or  to  the  patient.  These  may  include 
travelling  costs,  other  health  care  costs  and  the  cost  of  arranging  home  help  (Table  2). 
Direct  health  care  costs  include  those  directly  due  to  the  application  of  telemedicine  or 
the  conventional  alternative.  The  investment  costs  of  equipment  and  line  charges 
constitute  a great  deal  of  the  direct  (and  total)  costs  of  the  telemedicine  in  many 
specialties.  Both  PC  and  SC  units  must  invest  in  machinery,  software,  accommodation, 
and  networking.  In  economics,  these  costs  are  called  fixed  costs  (FC),  since  they  do  not 
vary  according  to  the  utilization  of  a produced  service.  In  addition,  fixed  costs  include 
the  monthly  rental  cost  of  the  network,  some  salaries  and  wages,  and  possibly 
administrative  expenses  of  the  hospital /centre.  Variable  costs  (VC)  include  costs  that 
vary  according  to  the  level  of  service  such  as  supplies,  drugs,  and  fees  for  service.  Since 
in  most  telemedicine  projects  health  care  professionals  are  working  only  occasionally  in 
the  project,  salaries  and  wages  can  be  counted  as  variable  costs. 

The  analysis  of  costs  is  commonly  undertaken  using  a one  year  period.  The  annual 
fixed  costs  are  calculated  by  dividing  the  investment  costs  by  the  utilization  time  of  the 
equipment  and  adding  the  other  annual  fixed  costs.  The  share  of  fixed  costs  per  patient 
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is  calculated  on  the  basis  of  the  annual  caseload.  If  the  variable  costs  do  not  change 
according  to  the  number  of  patients  per  year,  the  total  direct  health  care  costs  per 
patient  (TC)  is  calculated  by  adding  the  fixed  cost  and  variable  cost  per  patient. 

Indirect  costs  are  used  to  denote  the  time  of  patients  (or  their  families)  consumed  or 
freed  by  the  program  (Table  2).  The  intangible  costs  include  the  value  of  death,  pain 
and  HRQOL  (19).  They  are  valued  either  by  using  monetary  values  or  by  using 
economic  and  psychometric  scaling  techniques. 

Loss  in  working  time  (production)  can  be  estimated  on  the  basis  of  the  average  loss  in 
working  hours  which,  in  turn,  can  be  valued  according  to  the  mean  industrial  wages  to 
ensure  that  the  cost  estimates  have  not  been  affected  by  a few  extreme  values  or  by 
special  patient  groups.  Loss  of  leisure  hours  is  often  estimated  on  a time  basis  without 
monetary  valuation,  though  the  question  of  valuation  of  non-working  time  remains  a 
matter  of  debate.  Within  health  economics  there  is  a fundamental  debate  on  whether 
indirect  costs  should  be  included  in  analysis  of  costs.  The  discussion  is  centered  around 
how  to  cope  with  decreased  earnings  (losses  in  productivity).  Some  are  of  the  opinion 
that  they  should  be  excluded  from  the  agenda  altogether  (27).  Others  consider  that  they 
should  be  included  insofar  as  they  can  be  accurately  calculated  (45,  46). 

At  this  stage  in  the  assessment  of  telemedicine,  consideration  of  indirect  costs  has  not 
been  a major  factor.  However,  in  the  study  of  Halvorsen  and  Kristiansen  (36)  indirect 
costs  played  an  important  role  when  the  decision  rule  for  a teleradiology  service  was 
made.  Because  indirect  costs  may  have  substantial  distributional  effects  between 
different  subgroups  of  population,  they  can  be  shown  separately  by  natural  units  or 
monetary  values  and  the  decision  maker  may  include  or  exclude  them  from  the 
decision  process.  If  the  indirect  costs  are  included  in  the  total  costs  (TC)  of  the 
telemedicine  project,  there  is  a need  to  be  aware  that  there  is  a danger  of  double 
counting,  for  example,  if  production  losses  and  change  in  HRQOL  are  both  valued  by 
monetary  values  (19).  Monetary  valuations  for  HRQOL  in  telemedicine  studies  have 
yet  to  be  undertaken. 

The  costs  of  telemedicine  are  crucially  dependent  on  the  degree  of  utilization  of  the 
equipment.  For  instance,  if  the  primary  care  centre  is  able  to  use  video  conferencing  for 
many  types  of  consultations,  the  share  of  the  purchase  cost  of  the  equipment  (FC)  in  the 
total  costs  will  be  reduced.  In  principle,  a high  degree  of  equipment  utilization  (e.g., 
links  with  several  PC  centers  or  a large  number  of  applications)  at  an  SC  unit  offering 
telemedical  services  reduces  the  per  consultation  costs  of  telemedicine.  Much  depends, 
however,  on  the  number  of  personnel  required  to  produce  the  service  and  on  the  time 
used  for  telecommunication,  that  is,  on  the  variable  costs  of  a project.  As  a rule  of 
thumb,  the  shorter  the  distances  between  cooperative  parties,  the  higher  the  utilization 
of  the  equipment  should  be  to  make  the  investment  beneficial.  A good  example  is 
PACS  where  system  utilization  must  be  very  high  to  make  the  investment  realistic. 

These  questions  are  analyzed  in  the  framework  of  marginal  and  incremental  costs.  The 
marginal  cost  (or  benefit)  is  the  cost  (benefit)  borne  when  one  additional  unit  of  product 
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or  service  is  produced.  The  incremental  cost  (or  benefit)  refers  to  the  extra  cost  when  an 
additional  project  is  established  beside  the  old  ones  (19).  In  every  investment,  the 
marginal  benefits  should  exceed  the  marginal  costs  of  the  project  in  the  long  run  (34). 
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Economic  evaluation  methods 


Table  2 makes  reference  to  Effectiveness  Analysis,  Cost  Minimization  Analysis,  Cost- 
Effectiveness  Analysis  (CEA),  Cost-Utility  Analysis  (CUA)  and  Cost-Benefit  Analysis 
(CBA).  Basically,  the  costs  of  a project  are  evaluated  in  the  same  way  in  all  methods.  In 
cost  minimization  the  outcomes  are  assumed  to  be  equal  between  the  compared 
alternatives.  This  method  has  been  the  most  frequently  used  approach  in  the  field  of 
telemedicine.  Effectiveness  analysis  assumes  that  the  costs  of  the  alternatives  are  equal 
and  that  they  differ  only  in  their  effectiveness.  In  an  economic  evaluation,  at  least  two 
alternatives  should  be  compared.  If  the  costs  or  consequences  of  a new  technology  are 
evaluated  without  any  comparison  to  some  existing  alternative,  economic  interpretation 
of  the  results  in  terms  of  incremental  ratios  cannot  be  made. 

CEA  is  a form  of  economic  evaluation  where  the  costs  and  effectiveness  of  alternatives 
are  compared.  The  effectiveness  measures  used  in  the  analysis  include  both  natural 
unit  measures  (such  as  life  years  saved,  reduction  in  sick  leave  and  morbidity)  and 
HRQOL  measures  other  than  utility  measures.  CUA  is  a special  form  of  CEA  where  the 
consequences  are  measured  with  utility  measures,  that  is  QALY's  gained  (9, 19). 

The  results  of  CEA  and  CUA  are  shown  as  the  ratios  TC/E  and  TC/U  in  Table  2.  For 
meaningful  comparison,  it  is  important  to  analyze  the  additional  costs  that  one  program 
imposes  over  another,  compared  with  the  additional  effects,  benefits,  or  utilities  it 
delivers.  Incremental  cost-effectiveness  ratios  are  used  for  such  purposes  (19). 

TC/E  ratios  are  not  fully  comparable  between  studies.  In  principle,  the  different  QALY 
outcomes  are  assumed  to  be  comparable,  as  are  also  the  cost /QALY  ratios.  However, 
HRQOL  measures  and  different  valuation  methods  can  significantly  affect  the  magnitude 
of  QALY's  gained  by  interventions  as  well  as  to  the  ranking  order  of  the  ratios  of  the 
interventions  (57).  The  decision  rule  for  the  cost /QALY  ratios  is  that  the  lower  the  ratio, 
the  more  efficient  the  intervention.  The  QALYs  gained  should  usually  be  discounted  to 
their  present  value. 

The  outcomes  of  the  CBA  are  always  valued  by  monetary  terms  called  benefits.  The 
benefits  of  a health  care  technology  are  linked  to  the  resources  saved.  The  estimation  of 
direct  benefits  (cost  savings  in  the  health  care  sector  and  individual  direct  savings)  are 
relatively  easy  to  determine  and  value.  The  estimation  of  indirect  (production  losses)  and 
intangible  (HRQOL)  benefits  is  a more  complicated  process.  In  the  human  capital 
approach,  the  value  of  healthy  life  is  equal  to  the  discounted  future  earnings.  Another 
method  is  to  ask  patients  or  the  general  population  about  their  willingness-to-pay  for  a 
certain  change  in  HRQOL  and  the  risk  of  dying  over  time  (for  details,  see  Johansson  (41, 
42)  and  Thompson  (70). 

The  main  purpose  of  economic  analysis  is  to  give  decision  makers  information  on  which 
alternative  is  superior.  The  decision-making  criteria  for  CUA  and  CEA  are  based  on  a 
comparison  of  incremental  C/E  ratios  of  technologies:  the  lower  the  C/E  ratio,  the  better 
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its  efficiency  (9, 19).  CBA  yields  theoretically  the  best  results  for  resource  allocation,  if  all 
outcomes  can  be  realistically  valued  in  monetary  terms,  because  the  benefits  of  an 
intervention  show  directly  the  opportunity  costs  of  rejecting  it.  However,  the  QALY 
approach  also  involves  the  opportunity  cost  interpretation:  if  a certain  intervention  is 
chosen,  the  QALYs  in  the  next  best  alternative  are  lost  (18). 

Implementation  of  economic  analysis  to  telemedicine 

Although  the  basic  theory  of  economic  evaluation  is  reasonably  clear,  its 
implementation  to  telemedicine  is  less  certain.  Difficulties  can  be  found  in  the 
estimation  of  both  the  effectiveness  and  the  cost  side  of  the  analysis. 

It  can  be  difficult  to  establish  an  observable  and  empirical  link  between  telemedicine 
and  change  in  patient  outcome  (53).  The  implementation  of  the  service  may  change  the 
quality  and  process  of  care  and  access  to  care.  When  new  patients  who  may  have  a 
different  need  for  care  are  treated,  the  outcomes  of  the  projects  will  also  change  both  in 
the  conventional  alternative  to  telemedicine,  and  in  the  health  care  unit  as  a whole.  In 
addition,  the  valuation  of  both  health  and  non-health  outcomes  are  always  difficult. 

In  the  cost  side,  the  changing  technology  hinders  the  estimation  of  costs,  since 
successive  versions  of  equipment  have  different,  usually  decreasing,  prices.  The  same 
also  happens  in  telecommunication  technology.  As  McIntosh  and  Cairns  (53)  have 
pointed  out,  changes  in  access  to  care,  consultation  and  referral  patterns,  and  number 
and  skill  mix  of  staff,  may  have  significant  effects  on  costs.  Since  such  difficulties  are 
largely  connected  to  the  uncertainty  of  the  environment  in  which  telemedicine  is  used, 
these  questions  can  partly  be  addressed  by  undertaking  a sensitivity  analysis  of  study 
results,  which  is  considered  below. 

Finally  the  wide  implementation  of  telemedicine  and  new  information  and 
communication  technology  will  considerably  change  the  health  care  processes.  The  so 
called  welfare  cluster,  that  shows  the  importance  of  cooperation  between  different 
levels  of  health  care  together  with  other  related  sectors  (research  and  industry),  will 
change  the  costs  and  hopefully  the  outcomes  of  health  care  in  future  (21,  44). 
Transaction  cost  theory  has  yet  to  be  used  in  an  empirical  study  of  telemedicine; 
guidance  on  these  questions  is  not  yet  possible  (59). 

A useful  approach  when  taking  the  viewpoint  of  the  health  care  provider  is  "breakeven 
analysis"  which  considers  the  volume  of  consultations  needed  for  the  total  annual  costs 
of  the  two  types  of  services  (telemedicine  and  its  alternative)  to  be  equal.  The 
telemedicine  option  will  typically  have  higher  fixed  costs  because  of  equipment  and 
telecommunication  charges.  The  non-telemedicine  option  has  higher  variable  costs 
because  of  travel  and  other  time-related  expenditure.  After  a certain  number  of 
consultations,  the  costs  of  telemedicine  will  be  lower  than  those  of  providing  an 
alternative  type  of  service.  This  form  of  analysis,  which  is  based  on  a cost  minimisation 
approach,  has  been  used,  for  example,  in  assessments  of  teleradiology  (6)  and 
telepsychiatry  (17).  Figure  1 provides  an  illustration  of  this  approach. 
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Figure  1 : Breakeven  analysis  for  a telepsychiatry  application. 
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Source:  Reference  34 


Sensitivity  analysis 

Economic  evaluation  results  reflect  the  method  and  data  used  and  thus  the 
environment  in  which  they  are  derived.  Much  of  the  practical  interpretation  of  the 
results  revolves  around  sensitivity  analysis  in  which  cost  and  outcome  factors  are 
simulated  under  various  basic  assumptions.  In  this  way,  some  allowance  can  be  made 
for  future  developments.  The  four  broad  areas  of  uncertainty  in  the  analysis  relate  to 
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variability  in  sample  data;  generalizability  of  results;  extrapolation;  and  analytical 
methods  (53). 

In  the  assessment  of  telemedicine,  difficulties  can  be  found  in  all  of  these  areas.  For 
example,  available  data  may  not  be  representative  of  the  usual  patients  who  access  the 
service  and  how  well  they  comply  with  usual  practice  throughout  a whole  health 
system.  Extrapolation  is  based  mostly  on  assumptions  about  the  possible  relationship 
of  the  study  variables  in  the  real  world,  for  example,  the  cost  structure  of  the  clinic,  and 
the  patient  and  management  casemix  of  the  region.  In  addition,  the  inclusion  (or 
exclusion)  of  indirect  costs  or  other  cost  components  can  significantly  change  the  result 
and  interpretation  of  the  study. 

In  telemedicine,  sensitivity  analyses  could  be  done,  for  example,  by  location  of  primary 
care  unit,  number  of  patients,  case  mix,  useful  life  of  investments,  and  type  (price)  of 
investments.  Since  telemedicine  evolves  rapidly,  the  results  will  also  reflect  the  effects 
of  this  rapid  development  upon  investment  decisions.  Sensitivity  analysis  can  be 
helpful  in  formulating  recommendations  for  the  timing  of  procurement  of  certain 
telematic  services. 

Summaries  of  monetary  and  non-monetary  factors 

Any  realistic  appraisal  of  a telemedicine  application  will  need  to  consider  a range  of 
both  quantitative  and  qualitative  factors.  An  approach  which  has  been  suggested  for 
assessment  of  computed  radiography  applications  is  social  audit  analysis  (13).  With 
this  approach,  a matrix  of  data  on  monetary  items  plus  information  on  non-monetary 
benefits  is  produced.  Monetary  and  non-monetary  factors  in  different  areas  of  impact 
can  be  brought  together  for  comparison,  with  additional  detail  and  values  being 
included  as  assessment  results  become  available. 

Such  analysis  could  include  costs  and  benefits  to  specialists,  referring  physicians,  other 
health  care  professionals,  patients  and  their  families,  and  to  health  care  administrators 
and  funders.  Particular  benefits  which  apply  to  each  group  of  major  participants  in  the 
telemedicine  application  can  then  be  considered,  taking  care  to  avoid  double-counting 
of  benefits  achieved.  Consideration  can  be  given  to  providing  appropriate  weightings 
for  intangible  benefits,  in  association  with  those  for  which  monetary  valuations  are 
available.  Weightings  for  the  various  factors  would  take  into  account  the  environment, 
priorities  and  values  held  by  policy  makers  (33).  An  analogous  approach,  with  more 
emphasis  on  monetary  factors,  is  the  use  of  a cost-consequences  matrix  discussed  by 
McIntosh  and  Cairns  (53).  Table  3 illustrates  areas  of  impact  for  telepsychiatry  (33). 
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Table  3:  Summary  of  impact  for  a telepsychiatry  application. 
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A systematic  review  of  telemedicine  assessments 

The  field  of  telemedicine  is  currently  under  rapid  development  and  in  recent  years  a 
number  of  studies  have  addressed  the  feasibility  of  various  telemedicine  applications. 
Relatively  few  studies,  however,  have  been  able  to  collect  the  data  required  to  provide 
an  unambigious  demonstration  of  the  value  of  applications  (68).  Furthermore,  as  far  as 
we  are  aware,  there  have  so  far  been  no  attempts  to  gather  all  the  relevant  information 
about  the  effectiveness  of  telemedicine  together  in  the  form  of  a systematic  literature 
review,  though  a review  of  assessments  in  telemedicine-neuromedicine  was  included  in 
the  report  prepared  by  the  TASTE  project  (31).  Therefore,  we  decided  to  survey  the 
literature  in  a systematic  manner  to  clarify  the  current  state  of  the  art.  The  survey  is 
intended  to  help  decision-makers  who  are  under  commercial  or  public  pressure  to 
establish  telemedicine  services  to  get  an  objective  view  of  what  is  really  known  at 
present  about  the  effectiveness  and  cost-effectiveness  of  telemedicine. 

Literature  search  and  methods  for  the  systematic  review 

Details  of  the  search  strategy  are  shown  in  Appendix  A.  Use  of  this  approach  enabled 
us  to  identify  784  articles,  dealing  with  telemedicine,  that  were  published  between  1966 
and  November  1998. 

Initial  screening  of  the  identified  articles  was  based  on  their  abstracts.  All  abstracts 
were  read  by  two  of  the  authors  (AO  and  RR).  Selection  of  relevant  articles  was  based 
on  the  information  obtained  from  the  abstracts  and  was  agreed  upon  in  a consensus 
meeting  between  the  two  reviewers.  When  an  abstract  did  not  give  sufficiently  precise 
information  about  the  study  or  such  information  was  not  available  at  all,  the  article  was 
obtained  for  further  review. 

We  looked  for  articles  which  reported  outcomes  of  telemedicine  in  terms  of 
administrative  changes,  patient  outcomes  and  economic  assessment.  Articles  which 
were  limited  to  describing  the  feasibility  of  a certain  system,  the  technical  evaluation  of 
a system  (e.g.,  comparison  of  transmitted  radiological  images  with  plain  film  images, 
comparison  of  transmitted  histological  images  with  glass  slides)  were  excluded. 

Full-text  articles  obtained  for  closer  inspection  were  again  evaluated  by  the  two 
investigators  who,  in  a consensus  meeting,  made  the  final  decision  on  whether  or  not  an 
article  should  be  included  in  the  final  review,  using  the  criteria  given  above. 

Methodological  quality  of  the  included  studies 

The  strength  of  evidence  in  each  of  the  included  studies  other  than  those  concerned 
only  with  economic  analysis,  was  judged  according  to  the  classification  given  by  Jo  veil 
and  Navarro-Rubio  (43). 
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In  that  classification,  study  design  is  specified  on  the  following  nine  levels,  in 
descending  order  of  strength: 

I.  Meta-analyses  of  randomized  controlled  trials. 

II.  Large  sample  randomized  controlled  trials. 

III.  Small  sample  randomized  controlled  trials. 

IV.  Non-randomized  controlled  prospective  studies. 

V.  Non-randomized  controlled  retrospective  trials. 

VI.  Cohort  studies. 

VII.  Case-control  studies. 

VIII.  Non-contr oiled  clinical  series,  descriptive  studies,  consensus  methods. 

IX.  Anecdotes  or  case  reports. 

Each  level  is  further  qualified  by  conditions  of  scientific  rigor  for  the  study. 

For  the  purposes  of  the  review,  the  selected  assessment  studies  included  comparison 
between  a telemedicine  application  and  a control  group.  The  comparison  of 
telemedicine  and  some  level  of  conventional  alternatives  was  also  required  for  inclusion 
of  economic  analyses  in  the  review.  Articles  which  were  duplicates  of  the  same  authors' 
other  published  studies  were  excluded  - the  most  representative  of  the  studies  was 
included  for  further  consideration.  The  chosen  articles  were  described  using  the 
following  categories:  authors  and  name  of  the  study,  strength  of  evidence  according  to 
the  nine  level  classification,  objectives  of  the  study,  study  design,  setting  and  subjects, 
type  of  economic  analysis,  and  results  and  conclusion. 

Results 

One  hundred  full-text  articles  were  obtained  for  closer  inspection.  Of  these,  29  were 
deemed  to  fulfill  the  inclusion  criteria  of  the  review  and  are  listed  and  briefly  described 
in  Table  4.  Eighteen  of  the  articles  assessed  at  least  some  clinical  outcomes;  the 
remaining  11  were  mainly  economic  analyses.  Some  kind  of  economic  analysis  was 
included  in  19  (66%)  of  the  studies. 

The  economic  analyses  in  the  articles  were  mostly  variants  of  cost  analysis  (16  studies) 
(Table  4).  Cost-benefit  analysis  was  said  to  have  been  done  in  two  studies.  However 
they  were  methodologically  more  like  cost  analysis  studies,  since  the  benefits  were 
estimated  as  savings  (mainly  traveling  costs)  compared  to  the  conventional  alternative, 
that  is  the  costs  of  conventional  care.  Except  for  one  study,  the  analyses  measured  only 
direct  medical  costs  and  some  kind  of  estimation  of  transportation  costs  were  included 
in  14  studies.  Indirect  costs  were  assessed  in  one  study.  Incremental  cost  analysis  was 
also  performed  in  one  study.  Cost-effectiveness  ratios  were  calculated  in  two  studies. 
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In  all  of  these  studies,  effectiveness  was  defined  in  clinical  terms.  There  were  no  studies 
including  standardized  HRQOL  measures  or  QALY  calculations.  Some  kind  of 
discounting  of  costs  was  included  in  four  studies  and  there  were  seven  studies  with 
some  kind  of  sensitivity  analysis  of  the  study  results. 

Discussion 

According  to  the  results  of  the  review,  there  are  still  few  data  on  the  effectiveness  and 
cost-effectiveness  of  telemedicine.  Of  the  nearly  800  articles  surveyed,  most  were 
reports  on  feasibility  of  various  applications  and  only  a few  of  the  studies  reported  a 
controlled  comparison  of  a telemedicine  application  with  conventional  means  of 
providing  services.  More  than  half  of  those  assessed  clinical  outcomes;  the  rest  were 
economic  analyses. 

The  review  indicates  that  at  the  moment,  the  most  convincing  evidence  regarding  the 
effectiveness  of  telemedicine  deals  with  teleradiology,  teleneurosurgery,  telepsychiatry, 
and  transmission  of  echocardiographic  images.  Promising  results  have  also  been 
obtained  for  the  transmission  of  electocardiograms.  However,  even  in  these 
applications,  most  of  the  available  literature  refers  only  to  pilot  projects  and  short  term 
outcomes.  Most  other  telemedicine  applications  still  lack  scientific  evidence  regarding 
their  effectiveness. 

Economic  analyses  have  mostly  revealed  that  teleradiology,  especially  transmission  of 
CT-images,  can  be  cost-saving,  although  one  of  the  included  studies  did  not  find  this  to 
be  the  case  (36).  An  important  contribution  to  the  discussion  about  the  cost- 
effectiveness  of  teleradiology  is  the  study  by  Bergmo  (6)  which  explicitly  states  the 
workload  that  has  to  be  exceeded  in  order  to  achieve  cost-savings  by  using 
teleradiology.  A similar  study  by  Bergmo  has  shown  that  specialist  consultations  in  the 
field  of  otorhinolaryngology  can  be  performed  in  a cost-saving  way  when  the  work- 
load exceeds  a certain  number  of  patients  (5).  A pilot  project  on  telepsychiatry  used  a 
similar  approach  (17).  Such  studies  that  give  a clear  number  needed  to  treat  by  the 
telemedicine  option  are  helpful  for  decision  makers  when  faced  with  the  question  of 
whether  or  not  to  start  a new  telemedicine  service. 

Another  field  in  which  telemedicine  has  in  several  studies  been  shown  to  be  cost- 
effective  is  telepsychiatry.  For  applications  other  than  those  mentioned  above,  scientific 
data  on  effectiveness  and  cost-effectiveness  of  telemedicine  remains  limited. 

The  quality  of  the  economic  analysis  in  the  reviewed  papers  was  relatively  low,  with 
few  exceptions.  The  costs  included  varied  significantly  between  studies,  so  that 
comparison  of  the  cost  estimates  may  not  be  feasible  in  many  cases. 

There  were  also  several  economic  studies  that  did  not  give  detailed  information  about 
empirical  background  of  the  cost  and/or  benefits  included  in  the  calculations.  These 
studies  were  excluded  from  the  review.  For  example,  we  excluded  one  good 
teleradiology  cost-benefit  analysis  (38)  because  the  theoretically  good  economic  model 
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did  not  make  use  of  the  empirical  cost  and  benefit  estimations  made  at  the  specific  sites 
by  the  study  group. 

Cost-effectiveness  studies  in  telemedicine  are  still  very  scarce.  A systematic  comparison 
of  the  costs  and  more  work  on  the  effects  of  the  alternatives  should  be  done  in  the 
future.  Although  the  term  "cost-effectiveness"  was  frequently  used  in  the  studies,  the 
effectiveness  (and  sometimes  costs)  were  assumed  to  be  established  for  telemedicine 
without  any  scientific  evidence. 

More  than  four  years  ago  an  editorial  in  the  Lancet  stated  that  " although  much  is 
claimed,  the  economic  benefits  of  telemedicine  have  yet  to  be  proved"  (3).  Although  a 
limited  number  of  telemedicine  applications  have  up  to  now  been  shown  to  be  effective 
and  cost-effective,  that  conclusion  still  remains  valid  for  the  majority  of  suggested  ways 
to  utilize  telemedicine.  While  a number  of  detailed  studies  are  in  progress  in  several 
countries,  the  assessment  literature  has  yet  to  address  aspects  of  telemedicine 
applications  as  they  move  into  routine  use,  or  their  longer  term  impact  on  health  status, 
costs,  and  organization.  Other  dimensions  will  require  consideration  when  formulating 
approaches  to  further  economic  analysis.  These  will  include  sustainability  of  the 
telemedicine  service,  decisions  on  equipment  and  telecommunications,  impact  on  the 
overall  use  of  health  program  resources,  and  measurement  of  outcomes  (35). 

Consequently,  further  assessment  studies  in  the  field  of  telemedicine  are  still  clearly 
needed.  Decision  makers  under  public  and  commercial  pressure  to  start  new 
telemedicine  services,  should  link  implementation  of  new,  and  in  many  instances  costly 
technology  to  realistic  development  of  a business  case  and  subsequent  data  collection 
and  analysis. 
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Appendix  A:  Methodology  used  for  the  report 

Approaches  to  assessment  of  telemedicine 

The  present  study  of  telemedicine  assessment  draws  on  the  assessment  model 
developed  in  the  research  group  of  the  Health  Care  District  of  Northern  Finland  (56). 
The  project  has  received  financial  support  and  expert  assistance  from  FinOHTA.  In 
addition  to  the  earlier  research  report,  the  current  study  has  utilized  other  frameworks 
in  the  assessment  of  telemedicine  that  have  recently  been  published  (34,  53).  The 
appropriate  parts  of  the  CCOHTA  guidelines  for  economic  evaluation  of 
pharmaceuticals  (9)  have  been  used  to  update  the  general  assessment  framework  and 
the  literature  in  the  field. 

Literature  search  and  methods  for  the  systematic  review 

Computerized  literature  searches  were  performed  using  the  Medline  (1966-November 
1998),  Health  Star  (1975-October  1998),  Embase  (1988- August  1998),  and  CINALH 
(1982- August  1998)  databases  and  the  search  strategy  described  in  Appendix  A.  In 
addition  the  HSTAT-database  (Health  Services /Technology  Assessment  Text , The 
National  Library  of  Medicine),  the  Database  of  Abstracts  of  Reviews  of  Effectiveness 
(DARE,  NHS  Centre  for  Reviews  and  Dissemination),  the  NHS  Economic  Evaluation 
Database  and  the  Cochrane  controlled  trials  register  were  searched  using  the  search 
term  "telemedicine". 

The  search  strategy  used  was  as  follows: 

001  exp  telemedicine/ 

002  telemedicine. tw.  not  1 

003  telepsychiatry.tw.  not  1 

004  teleradiology.tw.  not  1 

005  teleconsult$.tw.  not  1 

006  or/1-5 

007  assess$.tw.  and  6 

008  evaluat$.tw.  and  6 

009  validat$.tw.  and  6 

010  feasib$.tw.  and  6 

011  pilot. tw.  and  6 

012  or/7-11 

013  12 
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